Papers with parallel-integrated method Parallel Context Windows
Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extending the maximum context lengths of language models are lacking a strong baseline for in-context few-shot classification and on more challenging Chain-of-Thought reasoning, such as HotpotQA, deteriorate question miscomprehension and false inference. |
| Approach: | They propose to harness window-wise attention and positional embedding techniques to extend the maximum context lengths of language models. |
| Outcome: | The proposed method is able to extend the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. |